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REFERENCE TO COMPUTER PROGRAM LISTING APPENDIX 

10 

Appendix A includes a printout of a computer program entitled "registration.cpp", 
"registration.h", and "regtable.m", which are incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

1 5 Field of the Invention 

The present invention generally relates to confocal microscopy. More particularly, the 
invention relates to increasing the scanning rate capability of confocal microscopes. 

Description of Related Art 

Confocal microscopy is a technique that allows visualization of small structures in light 

20 scattering material such as brain slices. It accomplishes this by combining point illumination with 
point detection. The point detection is achieved by using a pirJiole in an image plane that serves to 
filter light from out-of-focus planes above and below the area of interest thereby creating an optical 
section of a relatively thick specimen. 

The main limitation of confocal microscopes is the s{>eed of image acquisition, since every 

25 image is reconstructed on a point-by-point basis. Typical commercial systems, which rely on 
relatively slow galvanometer-driven mirrors to position the point illumination, have frame rates of 
approximately 1 Hz. Even the fastest systems, which scan several illumination spots 
simultaneously, can only record at approximately 200 Hz. One way that the slower systems are 
used for faster recording is by only collecting data from the pixels lying on a single line, but even 

30 this line-scan technique, which sacrifices flexibility in picking sites-of-interest, only boosts the 
effective frame rate to approximately 400 Hz. With the majority of current systems, faster imaging 
time is direcUy related to shorter dwell times at each site-of-interest, which reduces the achievable 
signal-to-noise ratio. To achieve the frame rate necessary for making functional recordings at 
several user-selected sites-of-interest, it is beneficial to have an addressable system that can 
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selectively visit several sites on a specimen without spending any time scanning over areas that do 
not contain structures of interest. 

The use of acousto-optic deflectors (AODs) can increase the speed at which the point 
illumination may be positioned and allows for random access scanning at user-selected sites-of- 
5 interest. However, the use of AODs necessitates a path of light returning from the specimen that is 
different than die illumination path and thus prevents die use of a stationary pinhole. This in turn 
requires a pinhole or filter that is spatially and temporally synchronized witii die scanning excitation 
spot. Although there are existing systems that utilize an AOD, tfiose systems only utilize an AOD 
to reposition the illumination point in one dimension. The deflection of die illumination point in die 

10 second dimension is accomplished by a relatively slow galvanometer-driven mirror such as one 
used on typical confocal microscopes. In addition, the existing systems diat utilize an AOD employ 
a slit in die direction diat die AOD deflects die illumination point, radier dian a pinhole, tiiereby 
preventing true confocal imaging. 

There exists, dierefore, a need for a confocal microscope that permits flexibility in selecting 

15 sites-of-interest widi increased scanning and recording rates for observing high-speed phenomena 
widiout reducing dwell time at each site-of-interest. Furdiermore, to enable accurate site selection, 
die same system should be able to collect fiill frame confocal images. 

SUMMARY OF THE PREFERRED EMBODIMENTS 

20 

In a preferred embodiment, the present invention comprises a random-access confocal 
microscope. Such a device is necessary for scanning only selected sites-of-interest in a specimen 
without the time requirements of scanning many sites and only using the results from die sites-of- 
interest. In order to achieve a faster sampling rate, it is advantageous to only scan selected sites-of- 

25 interest. Additionally, by only scanning at selected sites-of-interest, die dwell time at each site is 
much longer for a given frame rate dian widi a system diat must scan die entire field. Further, such 
high speed scanning is necessary to observe some phenomena. One example of such phenomena is 
signal processing and transmission in neurons, although die present invention will have useful 
application in other fields involving high-speed phenomena as well. 

30 The present microscope comprises a light source, a high-speed light deflector, a central 

processing unit (CPU), and an addressable spatial filter. The light source may be any collimated 
light source used for such a microscope, such as a laser. The high-speed light deflector preferably 
is an acousto-optic deflector (AOD); however, a spatial light modulator such as die digital 
micromirror device (DMD) from Texas bstruments may also be used. The AOD allows a higher 

35 proportion of die source light to be directed to die site-of-interest and thus is preferred. The AOD is 
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connected to the CPU, such that the CPU detennines where a beam of light from the light source is 
directed. The CPU may be any conventional processor that is capable of transmitting controlling 
signals to the high-speed light deflector and the addressable spatial filter. The addressable spatial 
filter is controlled by the CPU and is synchronized widi die high-speed light deflector to allow 
5 simultaneous illumination and detection of a site-of-interesL 

The addressable spatial filter may comprise a variety of arrangements that allow random- 
access detection of a point site-of-interest. The sites may be specified by a user after viewing a full 
frame confocal image of a specimen. The addressable spatial filter is not necessarily a physical 
pinhole, as conunonly used on previous confocal microscopes. In one embodiment, the addressable 

10 spatial filter is comprised of a DMD and a separate photodetector (such as a photodiode or 
photomultiplier tube). In a second embodiment, the addressable spatial filter is comprised of a 
complementary metal oxide semiconductor (CMOS) camera. The DMD provides an array of 
microscopic mirrors that can be actuated individually, allowing actuation of only mirrors 
corresponding to the location of the site-of-interest. The actuation of these mirrors will direct the 

15 returning fluorescence, reflection, or transmission of light from die sites-of-interest in the focal 
plane to the photodetector. Alternatively, a CMOS camera is capable of reading only designated 
pixels corresponding to sites-of-interest. Additionally, the CMOS camera allows individual pixel 
readout without the time delay of conventional imaging systems such as CCD cameras. Both the 
DMD and CMOS embodiments camera allow high-speed random access imaging of all sites-of- 

20 interest at greater than or equal to IkHz. 

In another embodiment, the present invention provides a method for acquiring optical 
recordings. The method comprises selecting at least one site-of-interest, configuring a high-speed 
light deflector to illuminate the at least one site-of-interest, configuring an addressable spatial filter 
to record the fluorescence, reflection or transmission of light from the at least one site-of-interest. 

25 and recording the light from the at least one site-of-interest. 

The method may further comprise sequentially selecting and illuminating a plurality of 
sites-of-interest. The method may still further comprise repeating the previous steps at a frequency 
greater than or equal to 500 Hz per frame. 

In still another alternate embodiment, the present invention comprises optical recordings 

30 created using the previously described apparatus and method for acquiring an image, 

BRIEF DESCRIPTION OF THE DRAWINGS 
For a more detailed understanding of the preferred embodiments, reference is made to die 
accompanying Figures, wherein: 
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Figure 1 is a schematic diagram of a confocal microscope constructed in accordance with a 
first embodiment of the present invention utilizing a DMD as an addressable spatial filter in 
conjunction with a separate photodetector; 

Figure 1 A is a schematic diagram of a DMD with mirrors in a first angular position that 
5 reflects light away from a photodetector and a second angular position that reflects light towards a 
photodetector; 

Figure 2 is a schematic diagram of a confocal microscope constructed in accordance with a 
second embodiment of the present invention embodiment utilizing a CMOS camera as bodi an 
addressable spatial filter and photodetector; 
10 Figure 3 is a schematic diagram of the electronic components in a first embodiment 

utilizing a DMD as an addressable spatial filter in conjunction with a separate photodetector; 

Figure 4 is a schematic diagram of the electronic components in a second embodiment 
utilizing a CMOS camera as both an addressable spatial filter and photodetector; 

Figure 5 is a schematic diagram of the optical components in a first embodiment utilizing a 
1 5 DMD as an addressable spatial filter in conjunction with a separate photodetector; and 

Figure 6 is a schematic diagram of the optical components in a second embodiment 
utilizing a CMOS camera as both an addressable spatial filter and photodetector. 

Figure 7 is a schematic diagram of die optical components in a third embodiment utilizing a 
DMD as both a high-speed light deflector and an addressable spatial filter. 
20 Figure 8 is a schematic diagram of optical components in a diird embodiment utilizing a 

DMD as both a high-speed light deflector and an addressable spatial filter. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Referring initially to Figure 1, a random access high-speed confocal microscope 5 includes 
25 a laser 10 that emits a light beam 15. Dght beam 15 is rapidly re-directed by an acousto-optic 
deflector 20. The new position of light beam 15 is shown in Figure 1 as a light beam 16, which is 
reflected by a beam splitter 50 (such as a dichroic mirror in die case of fluorescence) as a light beam 
17 onto a specimen 30. After light beam 16 is directed onto the specimen 30, a light beam 25 may 
be fluoresced, reflected, or transmitted by the specimen 30. The manner in which light beam 25 is 
30 produced will depend upon the composition of the specimen and any exogenous optical indicators 
that might be in use. light beam 25 from the specimen 30 passes through a beam splitter 50 (such 
as a dichroic mirror in the case of fluorescence) and to a digital micromirror device (DMD) 60, A 
central processing unit (CPU) 40 is connected to AOD 20 and sends an electronic signal 41 to 
control where light beam 15 from laser 10 is directed. CPU 40 also controls the angular position of 
35 individual micromirrors in DMD 60 by sending an electronic signal 45. Light beam 25 from a site- 

4 



wo 2004/038461 



PCTAUS2003/035441 



of-interest 35 is reflected by the DMD 60 as a light beam 27 to a photodetector 70. Site-of-interest 
35 preferably lies on focal plane 34 within specimen 30. 

AOD 20 allows for almost instant positioning of light beam 15 emitted from light source 10 
because AOD 20 does not have the inertia associated with typical galvanometer-driven mirrors used 
5 in conventional confocal microscopes. This increases die speed at which specimen 30 can be 
scanned for sites-of- interest 35. The scan rate is much higher than typical confocal microscopes, 
because both AOD 20 and DMD 60 do not have the inertia associated with conventional mirrors 
and can therefore move direcdy from one site to the next without scanning over intervening sites. 

Figure 1 A illustrates schematically how DMD 60 functions as an addressable spatial filter. 

10 DMD 60 is preferably an electro-opto-mechanical chip made by Texas Instruments and consists of 
an array of micromirrors 201, 202, and 203. Micromiirors 201-203 are in a first angular position 
210 unless a signal 45 is sent from CPU 40 causing one or more of the micromirrors to change to a 
second angular position 220, In Figure lA, micromirror 202 has been moved to angular position 
220 in response to a signal 45 sent by CPU 40. Micromirrors 201-203 are extremely small squares 

15 (approximately 16 jxm, or .000016 meters per side). This allows micromirrors 201-203 to change 
from first angular position 210 to second angular position 220 very quickly (approximately 20 ^s, 
or .000020 seconds). When micromirror 202 is in second angular position 220, light beam 25 from 
a site-of-interest 35 on specimen 30 is reflected as a light beam 27 to photodetector 70. In the 
process of focusing light beam 17 onto a site-of-interest 35 in Figure 1, some light shines on areas 

20 above and below the site-of-interest 35. These areas are illustrated as sites-of-non-interest 36 and 
38 in Figure lA. Micromirrors 201 and 203 remain in angular position 210 and reflect light beams 
26 and 28 from sites 36 and 38 that are not of interest. In angular position 220, micromirrors 201 
and 203 reflect light beams 27 and 28 away from photodetector 70. CPU 40 synchronizes DMD 60 
and AOD 20 so that only micromirror 202 corresponding to a site-of-interest 35 illuminated by light 

25 beam 17 (shown in Figure 1) will reflect a light beam 25 to the photodetector 70. The computer 
program listing appendix includes programs for synchronizing DMD 60 and AOD 20. 

Numerous sites-of-interest 35 may be selected and scanned sequentially while sampling all 
die sites-of-interest 35 at greater than or equal to 500 Hz. The sampling rate may be as low as the 
video rate of 20-30 Hz, but preferably it is higher, such as 3kHz, and more preferably 4kHz. Most 

30 preferably, the sampling rate is 25,000/n, where n is the number of sites-of-interest 35. Thus, for 6 
sites-of-interest 35, the sampling rate is 4.167 kHz. This number is based on die demonstrated 
ability of die present invention to sample a site-of-interest 35 every 40 \xs. Furtiiermore, the system 
is adaptive so that the number of sites studied simultaneously can be optimized to the type of signal. 
To study fast signals, fewer sites-of-interest 35 can be selected and to study slower signals, more 

35 sites-K)f-interest 35 can be simultaneously studied. 
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A second embodiment of the present invention is shown schematically in Figure 2. The 
random access high-speed confocal microscope 7 shown in Figure 2 utilizes a complementary metal 
oxide semiconductor (CMOS) camera 80 in place of the DMD 60 and photodetector 70 used in 
Figure 1. In Figure 2, laser 10 emits light beam 15, which is rapidly re-directed by AOD 20. The 

5 new position of light beam 15 is shown in Figure 2 as light beam 16, which is reflected by beam 
splitter 50 as light beam 17 onto specimen 30. Light beam 25 from a site-of-interest 35 on 
specimen 30 passes through beam splitter 50 and to CMOS camera 80. As in the first embodiment, 
central processing unit (CPU) 40 is connected to AOD 20 and sends electronic signal 41 to control 
where light beam 15 from laser 10 is directed. CMOS camera 80, which functions as an 

10 addressable spatial filter, is also connected to CPU 40. CMOS camera 80 is synchronized with 
AOD 20 to allow simultaneous illumination and detection of a site-of-interest 35 on specimen 30. 
Light beam 25 from specimen 30 is received by CMOS camera 80, which is comprised of multiple 
pixels 85, 86, and 88. CMOS camera 80 allows for individual pixel readout without the time delay 
of conventional imaging systems. CPU 40 sends an electronic signal 47 to CMOS camera 80 to 

15 read only pixels corresponding to a site-of-interest 35. Therefore, only pixel 85 that corresponds to 
light beam 25 from site-of-interest 35 will be read by CPU 40. Pixels 86 and 88, which correspond 
to light beams 26 and 28 from sites 36 and 38 that are not of interest, will be ignored by CMOS 
camera 80. Because only pixel 85 corresponding to site-of-interest 35 is read by CMOS camera 80, 
the rate at which specimen 30 may be scanned is increased. 

20 Figure 3 illustrates schematically a more detailed layout of electronic components utilized 

in one embodiment in which the addressable spatial filter comprises DMD 60 and a signal 
photodiode 71. This configuration is merely one example of numerous variations of electronic 
components that may be utilized in die present invention and is not intended to limit the scope of 
the present invention. In addition to CPU 40 and DMD 60, an electronics rack 100 is shown to 

25 include several components. CPU 40 contains a parallel port 260, a parallel port 270, a digital input 
/ output card 280, an analog-to-digital converter (ADC) controller 290, a digital-to-analog converter 
(DAC) 300, and a frame grabber 310. A video camera 320 is preferably connected to frame 
grabber 310. Video camera 320 is utilized for visualization of the specimen and for rough 
alignment of the components, while frame grabber 310 is used to display images from video camera 

30 320. Electronics rack 100 preferably contains a parallel port breakout 110, a digital input/output 
breakout 120, a DMD control 130, a trigger doubler 140, a voltage amplifier 150, an ADC 
converter 160, a multiplexer 170, a DAC breakout 170, and an analog signal conditioner 190. In 
addition to signal photodiode 71, there is a reference photodiode 75. The output of signal 
photodiode 71 is sent to a current-to-voltage converter 200, while the output of reference 

35 photodetector 75 is sent to a separate current-to-voltage converter 210. There are also two separate 
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AODs, 240 and 250, for deflection of the illumination beam (light beam 15 in Figure 1) in both the 
X- and y-axes. AOD 240 is controlled by AOD driver 220 and AOD 250 is controlled by AOD 
driver 230. Finally, a stepper motor controller 330 is used for controlling the position of the focal 
plane 34 within the specimen 30 (botfi shown in Figure 1). 
5 Various inputs and outputs of the components are illustrated in Hgure 3. Included below is 

a summary of the components and the functions served by each. DAC 300 is used to send 
addresses of the position of light beam 15 (shov^m in Figure 1) to AOD 240 and AOD 250. Analog 
signal conditioner 190 is used to make the voltage output range of DAC breakout 180 optimally 
match the necessary inputs for AOD driver 220 and AOD driver 230. Parallel port 260 sends 

10 addresses for all sites-of-interest 35 (shown in Figure 1) to DMD 60. Digital input / output 280 
controls cycling of the DAC addresses 1 80 and DMD 60 from one site-of-interest to the next site- 
of-interest and generates triggers for ADC 160. As illustrated in Figure 1, light beam 25 from 
specimen 30 is received by photodetector 70 (shown as photodiode 71 in Figure 3). In addition, 
noise from laser 10 is measured with reference photodetector 75. The output signal from signal 

15 photodiode 71 passes through current-to-voltage converter 200 and the output signal from reference 
photodetector 75 passes through current-to-voltage converter 210. The outputs of signal 
photodiode 71 and reference photodetector 75 are dien amplified by voltage amplifier 150. 
Multiplexer 170 (with sample and hold frmction capability) is then used to simultaneously sample 
signal photodiode 71 output and reference photodetector 75 output, which is then sent to ADC 160. 

20 Trigger doubler 140 is used to generate two ADC 160 triggers for each given pulse. ADC 160 uses 
the first trigger to digitize the output from signal photodiode 71 and the second trigger to digitize 
the output from reference photodetector 75 and then store the results in CPU 40. The signal 
photodetector output 70 can then be divided by the reference photodetector output 75 to remove the 
effects of noise from laser 10. Finally, parallel port 270 sends an output to stepper motor controller 

25 330 (used for focusing) and a reset of trigger doubler 140. 

Figure 4 illustrates schematically another example of the electronic components utilized in 
an embodiment incorporating a CMOS camera 80 in place of DMD 60 and signal photodiode 71 
(shown in Figure 3). This configuration is merely one example of numerous variations of electronic 
components that may be utilized in the present invention and is not intended to limit the scope of 

30 the present invention. Because the CMOS camera 80 does not need a separate photodetector, signal 
photodiode 71 is eliminated, as well as current-to- voltage converter 200. An additional difference 
between the components utilized in Figure 3 and Figure 4 is that CMOS camera 80 is controlled by 
digital input / output 265, instead of parallel port 260. All other electronic components shown in 
Figure 4 (and their functions) correspond to those described in Figure 3. 
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Figure 5 illustrates a view of the optical components utilized in another embodiment of the 
present invention incorporating an addressable spatial filter comprising DMD 60 and signal 
photodiode 71. In this figure, laser 10 emits light beam 15. A beam aligner 350 centers light beam 
15 before light beam 15 passes through beam expander 360 and AOD 240 and AOD 250. AOD 

5 240 and AOD 250 position light beam 15, and light beam 16 exits AOD 240 and 250. Beam 
aligner 370 then directs light beam 16 into demagnification bench 375, which controls the range of 
scanning for AOD 240 and AOD 250 and the final size of light beam 17 on specimen 30. Light 
beam 16 is directed to beam splitter 50, which reflects the short wavelength light beam 16 but 
passes the longer wavelength light beam 25 from the specimen 30. If the wavelengths of light beam 

10 16 and light beam 25 are tine same (such as when light beam 25 is reflected from the specimen 30 
rather than fluoresced), a polarizing filter and quarter wave plate (not shown) may be used in place 
of the beam splitter 50 to separate illumination beam 16 from reflected beam 25. 

Light beam 16 is reflected by beam splitter 50 as light beam 17, which is focused by 
objective lens 77 onto specimen 30. A portion 18 of light beam 16 passes through beam splitter 50 

15 and is used to measure fluctuations in the power output of laser 10 with reference photodiode 75. 
Light beam 25 from specimen 30 is collected by objective lens 77, passes through beam splitter 50 
and is received by DMD 60. As shown in Figure 1 A, light beam 25 is reflected off DMD 60 as 
light beam 27 to photodetector 70. In Hgure 5, a signal photodiode 71, which is used to make 
optical recordings, is shown as one example of a photodetector 70. In addition, a switch mirror 390 

20 can direct light beam 27 away from signal photodiode 71 and to a video camera 320, which can be 
used for visualization of specimen 30 and rough alignment of the components. Emission filters 380 
ensure that only the desired wavelengths of light beam 27 are detected. 

Hgure 6 represents a view of die optical components in an embodiment utilizing a CMOS 
camera 80 as both an addressable spatial filter and photodetector. In Figure 6, tht CMOS camera 

25 80 has replaced DMD 60, signal photodiode 71, and video camera 320. All otiier optical 
components remain identical to those found in Figure 5. 

Figure 7 represents a view of the optical components in an embodiment utilizing a DMD 60 
as both a high-speed Hght deflector and an addressable spatial filter. In Figure 7. the AODs 240 and 
250 (shown in Figure 5 and 6) have been eliminated because DMD 60 is now used to position light 

30 beam 17, which is focused by objective lens 77 onto specimen 30. In Hgure 7, laser 10 emits a 
light beam 15 via fiber optic cable 1 1 . Ught beam 15 first passes through a collimator 361 and then 
beam expander 360. Light beam 15 dien hits a first mirror 51 and is directed to a beam splitter 52, a 
third mirror 53, and DMD 60. A portion of the microscopic mirrors (not shown) in the DMD 60 are 
turned on so that some amount of die light from light beam 15 is re-directed as short-wavelength 

35 light beam 19 to die specimen 30 and a siie-of-interest 35. As best illustrated in Hgure 8, it should 
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be noted that mirror 53 is not on the same elevation as DMD 60 and specimen 30. Therefore, light 
beam 19 bypasses mirror 53. A portion 21 of light beam 19 is diverted by a beam splitter 76 and is 
used to measure fluctuations in the power of laser 10 with reference pholodiode 75. Light beam 25 
from specimen 30 also bypasses mirror 53 and is received by DMD 60. As shown in Figure lA, 
5 light beam 25 is reflected off DMD 60 as light beam 27. In Figure 7, light beam 27 is directed 
downward to mirror 53 (into the plane of the paper as drawn Egure 8) and then reflected so that it 
passes through beam splitter 52 and to a photodetector. In Rgure 7, a signal photodiode 71, which 
is used to make optical recordings, is shown as one example of a photodetector 70. In addition, a 
switch miiTor 390 can direct light beam 27 away from the signal photodiode 71 and to a video 
10 camera 320, which can be used for visualization of the specimen and rough alignment of the 
components. Emission filters 380 ensure that only the desired wavelengths of light beam 27 are 
detected. 

Figure 8 represents a front view of DMD 60, mirror 53, and specimen 30 (as shown in 
Figure 7). From this view, it is clear that DMD 60 and specimen 30 are arranged so that light 

1 5 beams 19 and 25 bypass mirror 53 and do not pass through mirror 53. 

The above discussion and Figures are meant to be illustrative of the principles and various 
embodiments of the present invention. Numerous variations and modifications will become 
apparent to those skilled in the art once die above disclosure is fully appreciated. For example, by 
placing the point detection components (i.e. CMOS 80 or DMD 60 and photodetector 70) on the 

20 side opposite the specimen 30 from the illumination beam of light 17, the amount of light 
transmitted and absorbed by the specimen 30 may be observed. The present invention may also be 
used in aspects of high-speed imaging other than signal processing and transmission in neurons. It 
is intended tiiat the following claims be interpreted to embrace all such variations and 
modifications. Sequential recitation of steps in the claims is not intended to require that the steps be 

25 performed sequentially, or that one step be completed before conmiencement of anotiier step. 

The present disclosure hereby incorporates by reference U.S. Patent 5,587,832 {Krause\ 
U.S. Patent 4,893,008 (Horikawa\ U.S. Patent 4,863,226 (Houpt et al\ U.S. Patent 4,662,746 
(Hombeck), U.S. Patent 6,084,229 (Pace et al,\ and U.S. Patent 4,827,125 {Goldstein) in tiieir 
entirety, except to the extent they conflict with the present disclosure. 

30 The present disclosure also hereby incorporates by reference, except to the extent that it 

conflicts with the present disclosure, the paper entitled "A High-Speed Confocal Laser-Scanning 
Microscope Based on Acousto-Optic Deflectors and a Digital Micromirror Device" by V. Bansal, 
S. Patel, P. Saggau. This paper was presented and published at the 25^ Annual International 
Conference of the IEEE Engineering in Medicine and Biology Society (held September 17-21, 

35 2003). 



